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Prefatory  Note 

This  paper  was  presented  by  Dr.  Boyles  as  part  of  a  tri¬ 
service  symposium  entitled,  Development  of  a  Combat  Aviator 
Criterion.  Other  contributors  to  the  symposium  represented  the 
Naval  Aerospace  Medical  Institute,  Pensacola,  Florida,  and  the 
U.S.  Air  Force  Human  Resources  Laboratory. 

This  paper  describes  research  performed  by  Division  No.  6 
(Aviation),  of  the  Human  Resources  Research  Organization,  at 
Fort  Rucker,  Alabama,  under  Work  Unit  PREDICT,  Correlational 
Analysis  of  Aviator  Performance.  The  paper  deals  with  one 
aspect  of  the  Work  Unit,  combat  aviator  criterion  development. 


COMSAT  AVIATOR  CRITERION  DEVELOPMENT 

Wiley  R.  Boyles,  Peter  R.  Prunkl, 
and  James  L.  Wahlberg 


Many  different  methods  for  evaluating  the  effectiveness  of  human 
work  behavior  are  found  in  the  professional  literature.  Two  of  the 
m. jor  methods  are  objective  measures  of  performance— in  ter^s  of  what 
Dunnette  (1)  calls  "organizational  consequences"— and  ratings  by  persons 
who  have  had  opportunity  to  observe  worker  performance.  Of  the  rating 
methods,  graphic  rating  scales  are  most  common;  others  include  critical- 
incident  reports,  forced-choice  techniques,  free-written  ratings,  nomi¬ 
nations  (by  peers,  subordinates,  and  superiors),  ranking  methods, 
checklists,  and  work-sample  tests  (2).  Appropriateness  of  technique  is 
a  function  of  the  complexity  of  the  work,  the  types  of  raters  available, 
die  structure  of  the  organization  where  the  evaluations  are  to  be  used, 
the  purpose  of  the  evaluations,  and  to  use  Dunnette 's  (1)  terms  again, 
the  "situational  and  social  circumstances"  of  the  job  environment. 

In  relating  "appropriateness"  to  the  Army  aviation  combat  situation, 
we  are  faced  with  the  necessity  of  evaluating  performance  on  highly  com¬ 
plex  jobs  in  work  situations  in  which  many  types  of  raters  are  available. 
Traditionally,  the  Army  has  used  ratings  by  superiors  as  a  means  of 
evaluating  job  performance  in  operational  situations.  We  are  interested 
in  obtaining  evaluations  of  job  performance  for  purposes  of  research, 
intending  to  use  them  for  validation  of  selection  and  training  tech¬ 
niques.  Also,  the  validation  must  be  in  terms  of  multiple  criteria, 
for  our  aviators  must  be  effective  officers  as  well  as  effective  pilots. 
The  situational  and  social  circumstances  of  the  jobs  range  widely. 
Objective  performance  measures,  in  terms  of  organizational  consequences, 
are  difficult  to  collect. 

The  Army's  Behavioral  Science  Research  Laboratory  has  launched  a 
systematic  effort  to  establish  criteria  for  officer  combat  performance. 
However,  no  large-scale  Army  effort  to  evaluate  potential  criteria  of 
aviator  performance  in  combat  now  exists.  HumRRO  has  taken  some  pre¬ 
liminary  steps  in  this  direction,  and  further  steps  are  planned.  We 
welcome  technical  suggestions.  We  have  surveyed  the  literature,  and 
have  some  data  that  show  promise,  but  at  this  time  we  have  many  more 
questions  than  answers. 

At  the  HumRRO  Aviation  Division,  we  are  constructing  a  data  bank 
for  Army  aviators.  We  are  working  toward  the  establishment  of  a  multi¬ 
variate  system  for  prediction  of  performance  in  both  training  and  opera¬ 
tional  situations.  Criteria  of  training  success  are  relatively  easy  to 
establish;  criteria  of  individual  adequacy  in  Army  aviation  combat 
obviously  are  not. 


Our  research  has  included,  over  the  past  two  years,  the  administra¬ 
tion  of  paper-and-pencil  attitude  measures  developed  in  earlier  HumRRO 
research  on  motivation  under  combat  stress.  Because  these  inventories 
seemed  to  have  promise  for  aviation  training  and  secondary  selection 
situations,  we  have  administered  them  to  several  thousand  potential 
aviators.  Our  results  show  that  the  inventories  differentiate  between 
aviator  candidates  and  other  members  of  the  basic  combat  trainee  popula¬ 
tion,  and  between  aviation  trainees  who  successfully  complete  flight 
training  and  those  who  fail  because  of  flying  deficiencies. 

Concurrently  with  administration  of  these  inventories,  we  began 
exploratory  evaluations  of  problems  in  combat  criterion  development. 
Rickus  and  Berkshire  (3)  discussed  in  1968  the  problems  encountered 
during  past  efforts  to  evaluate  the  performance  of  the  combat  aviator. 
Some  of  these  problems  are  results  of  the  complexity  of  combat;  others 
are  functions  of  the  complexity  of  aviation. 

In  our  first  step  toward  deriving  criteria  for  combat  validation  of 
the  data  bank  material,  the  critical  incident  technique  to  collect 
samples  of  ineffective  behavior  was  used.  This  research  was  reported 
in  1968  (4).  Among  the  results  was  a  finding  similar  to  that  of  the 
Navy;  refusal  of  a  pilot  to  fly  with  another  aviator  is  a  frequent 
identifying  incident  connected  with  perceived  ineffectiveness .  The 
results  also  indicated  that  a  manageably  small  number  of  behavior  cate¬ 
gories  accounted  for  a  large  portion  of  the  critical  incidents. 

Building  on  these  results,  we  have  recently  begun  to  collect  data 
with  an  aviator  combat  performance  rating  instrument  that  includes  peer 
nominations,  a  number  of  five-point  graphic  rating  scales,  and  some 
dichotomous  items.  Some  of  the  scales  and  items  were  based  on  our 
critical  incident  data,  and  some  were  based  on  subsequent  interviews 
with  veteran  combat  aviators.  Others  were  taken  fror  instruments 
developed  by  Roff  (5)  and  by  Rickus  and  Berkshire  (3).  An  Appendix 
to  this  paper  contains  the  nomination  format,  rating  scales,  and  dicho¬ 
tomous  items  chosen  from  a  pretest.  In  the  Rickus  and  Berkshire  study, 
Navy  aviators  agreed  that  the  flight  surgeon  was  in  the  best  position 
to  rate  their  combat  performance.  However,  in  our  work,  interviews, 
questionnaires,  and  consensus  of  aviators  indicated  that  fellow  aviators 
are  in  the  best  position  to  evaluate  Army  aviator  combat  performance. 

We  are  assessing  the  psychometric  properties  of  f'ur  items  in  terms  of 
their  ability  to  distinguish  between  aviators  nominated  as  "most  profi¬ 
cient"  or  "least  proficient"  by  peers,  subordinates,  and  superiors. 

Since  there  are  two  distinct  categories  of  Army  aviators— warrant  offi¬ 
cers  and  commissioned  officers— we  are  exploring  differences  in  ratings 
as  a  function  of  category  interrelationships.  Additional  possible 
moderator  variables  included  are  aviation  unit  size,  unit  mission, 
terrain  types  over  which  an  aviation  unit  operates  (flying  behavior 
requirements  range  widely  as  a  function  of  terrain),  and  frequency 
of  observation. 

In  general,  our  scales  deal  with  quality  and  speed  of  combat  deci¬ 
sions,  flexibility,  combat  flying  content  knowledge,  risk-taking 
behavior,  confidence,  leadership,  endurance,  and  emotional  stability. 
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One  objective  performance  scale  item,  relating  to  selection  by  superior 
for  tactical  emergency  missions,  was  included.  The  dichotomous  items 
call  for  some  responses  in  terms  of  objective  performance  measures  and 
others  in  terms  of  subjective  evaluations.  As  would  be  expected  from 
their  content,  almost  all  produce  significant  x2s  between  least  profi¬ 
cient  and  most  proficient  nominees.  The  two  exceptions  are  "turned  in 
wings"  and  "killed  in  action"  which  are  events  too  rare  to  produce 
significance.  The  proportion  of  wounded  in  action  is  significantly 
higher  for  those  aviators  nominated  as  "most  proficient" 

{x2  =  17.60,  p< . 001 } . 

As  in  the  1968  Rickus  and  Berkshire  study  (3),  the  most  frequently 
cited  identifying  behavior  for  low  proficiency  aviators  was  "people 
refuse  to  fly  with  him."  This  was  cited  for  61%  of  the  aviators 
nominated  as  "least  proficient,"  while  reported  for  less  than  1%  of 
those  named  "most  proficient."  On  the  other  hand,  the  item  which 
appeared  with  second  highest  frequency  in  the  Rickus  and  Berkshire 
study,  "turned  in  his  wings,"  was  encountered  in  only  two  of  201  cases 
in  our  recently  collected  data.  The  item  on  the  HumRRO  scale  that 
appeared  with  second  highest  frequency  among  least  proficient  aviators 
was  item  26,  "Did  he  react  to  loud  noises  in  the  aircraft  as  if  they 
were  gunshots?"  While  50%  of  the  least  proficient  aviators  were 
reported  to  have  reacted  in  this  manner,  20%  of  the  most  proficient 
aviators  were  also  reported  to  have  reacted  this  way.  Thus,  while 
this  item  was  significant  at  the  .001  level,  there  were  other  items 
that  were  better  discriminators  between  most  and  least  proficient 
aviators.  For  example,  47%  of  the  least  proficient  aviators  were  said 
to  panic  in  "hot"  situations,  whereas  less  than  1%  of  the  most  profi¬ 
cient  aviators  did  so. 

Let  me  note  that  we  asked  this  sample  for  nominations  of  aviators 
"least  proficient  while  flying  under  enemy  fire,"  rather  than  for 
"ineffective"  or  "unsatisfactory"  aviators.  We  are  dependent  for 
definition  of  a  satisfactory-unsatisfactory  dichotomy  on  responses  to 
Item  22  of  our  scale.  Item  22  results  showed  55  of  201  aviators  nomi¬ 
nated  as  "least  proficient"  were  reported  as  having  been  removed  from 
the  job  because  of  poor  performance. 

Of  our  situational  context  variables,  almost  all  proved  to  be 
significant  moderators  of  one  or  more  items.  This  fact  warrants 
early  cross-validation  of  results. 

Asymmetric  moderation  was  obvious  in  the  data.  For  example,  we 
found  no  significant  difference  between  the  frequencies  with  which 
warrant  officers  and  commissioned  officers  were  named  "least  proficient" 
as  a  function  of  the  rater's  category.  That  is,  commissioned  officers 
seemed  no  more  prone  to  name  warrant  officers  "least  proficient"  than 
did  the  warrant  officers  themselves.  On  the  "most  proficient"  item, 
however,  the  situation  was  quite  different.  Warrant  officers  more 
frequently  named  warrant  officers  "most  proficient;"  commissioned 
officers  named  commissioned  officers  {x2  *  12.00,  p<.001}.  This  sort 
of  finding  emphasizes  the  necessity  for  studying  all  possible  moderator 
combinations  in  the  data. 
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Apart  from  the  rating  scale  data,  two  other  sources  offered 
information  on  our  aviators:  a  Background  Activities  Inventory  (BAI), 
the  previously  mentioned  attitude  inventory  designed  from  a  model  of 
reaction  to  combat  stress,  and  a  Vietnam  Returnee  Questionnaire  admin¬ 
istered  to  all  aviators  returning  to  Fort  Rucker  from  Vietnam. 

From  the  BAI,  we  found  that  age  upon  entering  the  flight  program 
was  significantly  higher  for  those  aviators  named  "most  proficient" 

{t  =  2.40,  p< . 05 ;  r  ^  =  .40,  p<.01}. 

The  mean  age  was  21.2  years  for  this  group  as  compared  to  19.9  years 
for  the  "least  proficient"  group. 

These  results  were  not  a  function  of  differences  in  amount  of  com¬ 
bat  experience.  The  47  pilots  in  this  sample  had  one  tour  in  Vietnam 
as  aviators  and  had  approximately  the  same  amount  of  experience. 

The  BAI  also  furnished  information  regarding  past  participation  in 
dangerous  sports  and  activities.  We  hypothesized  that  our  pilots  rated 
"most  proficient"  would  have  participated  more  often  in  these  activi¬ 
ties  than  those  raed  "least  proficient."  Our  data  did  not  support 
this  hypothesis:  The  most  proficient  aviators  had  significantly  lower 
exposure  to  these  sports  and  activities  than  the  least  proficient  pilots 

{i  =  2.60,  p<.0S;  z»  ^  *  -.35,  p<.05). 

From  our  second  additional  data  source,  the  Vietnam  Returnee  Ques¬ 
tionnaire,  various  self-report  items  completed  by  21  of  our  ratees  were 
selected.  No  items  showed  a  significant  difference  between  the  two 
groups.  There  was  no  significant  difference  in  self-reports  of  days 
flown  in  an  average  month,  number  of  accidents,  number  of  forced  land¬ 
ings,  and  number  of  times  grounded. 

I  have  given  a  rather  cursory  look  at  the  details  of  our  results, 
as  this  presentation  is  intended  to  be  more  provocative  than  documen¬ 
tary,  I  think  we  are  getting  some  thought-provoking  data.  In  answer 
to  "Where  do  we  go  from  here?"  I  will  list  some  of  the  alternatives 
under  consideration.  One  is  a  refinement  of  our  scales  using  the 
method  developed  by  Smith  and  Kendall  (6),  which  seems  to  have  great 
potential  for  optimizing  the  combination  of  the  components  of  psycho¬ 
metric  soundness  on  the  one  hand,  and  user  acceptability  and  accuracy 
on  the  other.  That  method  proceeds  from  critical  incident  collectings, 
categorized  by  major  qualities  necessary  for  job  success,  through  inde¬ 
pendent  matching  of  qualities  and  incidents,  to  independent  judgmental 
locations  of  the  instruments  on  a  scale,  with  all  steps  performed  by 
content  experts  and  with  vigorous  requirements  for  acceptability  of 
both  the  behavior  descriptions  making  up  the  scales  and  qualities 
measured  by  the  scales. 

Another  possibility  is  exploration  of  the  utility  of  the  multitrait, 
multirater  method,  either  in  combination  with  or  as  an  alternative  to 
the  Smith  and  Kendall  (6)  approach.  The  utility  of  this  approach,  as 
described  by  Lawler  (7),  seems  to  hold  up  well  across  a  variety  of 
complex  jobs. 


4 


Finally,  I  would  like  to  emphasize  the  continuing  importance  of 
systematic  pan-service  coordination  of  behavioral  studies.  Both 
significant  overlap  and  important  differences  exist  among  critical 
job  requirements  for  Navy,  Marine,  Air  Force,  Coast  Guard,  and  Army 
aviators.  Comparison  of  our  preliminary  data  in  this  area  with  those 
of  Rickus  and  Berkshire  (3)  shows  both  strong  convergence  and  strong 
divergence.  Knowledge  of  the  commonalities  and  disparities  in  our 
aviator  job  requirements  could  have  a  synergistic  effect. 
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Appendix 

COMBAT  RATING  SCALE 


A.  UNIT  INFORMATION 

1.  Your  name  and  grade: _ _ 

2.  Name  the  unit  with  which  you  served  the  longest  during  your 

last  RVN  tour. _ 

3.  Name  the  date  that  you  reported  to  this  unit. _ 

4.  Name  the  date  that  you  left  this  unit. _ 

5.  How  many  aviators  were  assigned  to  this  unit  during  your  tour? 


6.  Primarily,  what  type  of  mission(s)  did  you  fly  in  this  unit? 


B.  AVIATOR  INFORMATION 

1.  Think  of  the  aviators  in  your  unit  who  you  have  observed  fre¬ 
quently  under  enemy  fire.  Name  three  of  these  men  who  you 
consider  most  proficient  in  their  performance  while  under  fire. 
(Use  1  as  most  proficient,  2  as  second  most  proficient,  etc. 
Feel  free  to  use  any  aviators  in  your  unit— superiors,  sub¬ 
ordinates,  or  any  others— living  or  dead. ) 

1. _ _ _ 

2. _ 

3. _ 

2.  Name  the  three  aviators  you  consider  the  least  proficient  while 
flying  under  enemy  fire.  (Use  1  as  least  proficient,  2  as 
second  least  proficient,  etc.  Again  you  may  use  any  aviators 
in  your  unit-superiors,  subordinates,  or  any  others— living 

or  dead.) 

1. _ 

2. _ 

3. _ 

C.  CHARACTERISTIC  INFORMATION 

You  have  just  rated  six  people  in  terms  of  their  flying  proficiency 
in  combst  situations.  Now  rate  each  of  the  six  aviators  you  have 
named  in  terms  of  the  following  characteristics. 
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Remember  that  you  are  rating  each  person  at  a  time  on  each  char¬ 
acteristic.  A  good  combat  aviator  may  have  some  bad  points  and 
a  bad  aviator  may  have  several  good  points  while  flying  under 
enemy  fire. 

RECORD  YOUR  ANSWERS  ON  THE  ANSWER  SHEETS.  Read  each  characteristic 
carefully.  A  is  the  extreme  and  corresponds  to  the  behavior  directly 
below  A.  E  is  the  opposite  extreme  and  its  description  is  also 
below  the  letter.  Consider  C  to  be  the  average  behavior.  In  some 
cases  the  behavior  characteristic  for  C  is  written  below  it. 


RATING  SCALE 


A 

1.  Decisions  al¬ 
ways  worked 
when  under 
enemy  fire 

B 

C 

D 

E 

Decisions  never 
worked  when  under 
enemy  fire 

A 

B 

C 

D 

E 

2.  Indecisive  in 
his  combat 
decisions 

Reached  a  deci¬ 
sion  rapidly 
when  under  fire 

A 

B 

c 

D 

E 

3.  Flew  mechani¬ 
cally  in  "hot" 
situations 

Adjusted  to  the 
changing  tactical 
situations 

A 

B 

c 

D 

E 

4.  Had  learned 
the  tricks  of 
combat  flying 

Little  knowledge 
of  combat  flying 

A 

B 

c 

D 

E 

5.  Excellent  fly¬ 
ing  judgment 
while  under 
-ire 

Extremely  poor 
judgment  while 
under  fire 

A 

B 

c 

D 

E 

6.  Remembered  his 
training  in 
"hot"  situa¬ 
tions 

Forgot~important 
procedures  while 
under  enemy  fire 

A 

7.  Took  no  chances 
while  in  combat 

B 

£ 

D 

E 

Took  many  chances 
while  under  fire 

A 

8.  Calm  and  re¬ 
laxed  under 
fire 

B 

C 

D 

E 

Extremely  nervous 
and  tense  under 
fire 

A 

B 

C 

D 

E 

9.  Not  confident 
about  his  abil¬ 
ity  when  under 
fire 

Confident,  he 
felt  he  could 
handle  any 
situation 

Overconfident 
about  his  ability 
when  under  fire 

10.  A 

B 

f* 

\j 

D 

E 

Was  unwilling 
to  fly  tactical 
emergency 
missions 

Was  willing  to 
fly  the  tactical 
emergency  missions 

A 

B 

C 

D 

E 

11.  Was  usually 
picked  by 
superior 
for  tactical 
emergency 
missions 

Was  never  picked 
by  superior  for 
tactical  emer¬ 
gency  missions 

A 

B 

C 

D 

E 

12.  Practiced  the 
principles  of 
good  leader¬ 
ship 

Never  practiced 
the  principles 
of  good  leader¬ 
ship 

A 

B 

c 

D 

E 

13.  Sought  out 
leadership 
situations 

Avoided  leader¬ 
ship  situations 

A 

B 

c 

D 

E 

14.  Did  not  do 
his  job  when 
under  file 

Did  his  job 
and  nothing 
more  when 
under  fire 

Did  more  than 
what  was  expected 
of  him  when  under 
fire 

A 

B 

c 

D 

E 

IS.  When  under  enemy 
fire  flew  at 
unnecessarily 
high  altitude 

Flew  at  appro¬ 
priate  altitude 
when  under  fire 

Flew  unnecessarily 
close  to  the  tar¬ 
get  when  under 
fire 

A 

B 

C 

D 

E 

16.  Always  kept 
a  clear  head 
under  fire 

"Lost  his  head" 
in  combat  situa- 
situations 
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B 


C 


A 

17.  Fatigued  easily 
when  under  fire 


Had  great  endur¬ 
ance  under  fire 


ABC 

18.  Evidenced  feel¬ 
ings  of  fear 
before  diffi¬ 
cult  missions 


Never  showed  his 
feelings  of  fear 
before  difficult 
missions 


D.  SUMMARY  INFORMATION 

19.  Did  he  ever  turn  his  wings  in? 

20.  Did  he  ever  want  to  turn  his  wings  in? 

21.  Did  people  ever  refuse  to  fly  with  him? 

22.  Was  he  given  nonflying  duties  because  of  poor  performance? 

23.  Did  he  turn  down  aircraft  that  other  aviators  felt  were 
flyable? 

24.  Was  he  ever  charged  with  one  or  more  accidents? 

25.  Did  he  ever  appear  before  a  flight  evaluation  board? 

26.  Did  he  react  to  loud  noises  in  the  aircraft  as  if  they  were 
gunshots? 

27.  Did  he  claim  to  be  sick  in  order  to  keep  from  flying? 

28.  Did  he  ever  abort  a  mission  unnecessarily? 

29.  Did  he  panic  in  "hot"  situations? 

30.  Did  he  frequently  downgrade  Army  aviation? 

31.  Was  he  ever  wounded  in  action? 

32.  Was  he  killed  in  action? 
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